perf(gfql): seeded typed-hop fast path — seeded 1-hop RETURN in ≤1ms (#1755)#1759
Merged
Conversation
lmeyerov
commented
Jul 21, 2026
lmeyerov
commented
Jul 21, 2026
lmeyerov
commented
Jul 21, 2026
… tax (#1755 lever-3) The degenerate-shape fast path (_try_chain_fast_path) previously (a) bailed on any typed edge (edge_match) and (b) for constrained shapes still validated BOTH edge endpoints against the FULL node table before applying the seed — so a seeded 1-hop paid O(E) object/isin scans over every edge. This is the root of the #1755 seeded- Cypher "abstraction tax". Changes (pandas + cuDF, engine-agnostic): - Accept typed edges: edge_match is a plain edge-frame filter (filter_by_dict), applied on the reduced frontier — same result set as the full hop. - Seed-first: when a node filter is present, reduce edges by the from-side seed BEFORE the typed-edge scan and endpoint validation, and apply the destination filter to the SMALL gathered dst nodes rather than the full node table. Endpoint validation + result-node build then run on the tiny frontier (small isin key -> small hashtable, no O(E)-values scan), so a seeded 1-hop is O(result) not O(E). Parity byte-identical fast-vs-full across unconstrained/seeded/dst-filtered/reverse typed hops (test_chain.py differential + gating updated: edge_match moves from a BYPASS shape to a FAST shape). Measured pandas (50k/200k seeded typed 1-hop, warm- median): 47.0 -> 5.5 ms (8.6x), parity-exact. mypy clean, all chain tests pass. NOTE: prototype on the bench branch (the fast-path infra _try_chain_fast_path is itself not yet on master); lands stacked on that infra. Remaining to the <=1ms gate: a pandas numpy-level specialization (ideal hand-probe = 0.911ms) and/or the #1658 index for scale-invariance. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
…1ms (#1755 lever-3) Adds _seeded_typed_hop_numpy_pandas: a pandas-only numpy specialization of the seeded typed 1-hop, gated to scalar node/edge filters + directed (falls back to the engine-agnostic seeded branch for predicates/undirected/cuDF). Seed-reduces edges before the object-typed edge_match scan, applies the destination filter to the small gathered dst nodes, and does a SINGLE full-node scan to gather result nodes — so a seeded lookup is a few numpy passes, not O(E) pandas machinery. Measured dgx (26.02-gfql-polars, 50k/200k, native [n({id}),e(edge_match),n({type})], warm-median): full path 29.10 ms -> fast path 0.553 ms (52.6x), byte-identical parity. GATE MET (<=1ms) on the NATIVE seeded typed hop. All 57 chain tests pass, mypy clean. NOTE: this closes the native surface only. The cypher string g.gfql("MATCH (m {id})-[:T]->(p) RETURN p") lowers to NAMED nodes (m,p) + label__ filters + a 4th ASTCall projection, so it does not yet engage this fast path (63 ms). Closing the cypher surface = the next lever (named-node + label + lean projection). Prototype on the bench branch (fast-path infra not yet on master). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
…<=1ms (#1755) Closes the #1755 seeded-Cypher abstraction tax on the CYPHER STRING surface. g.gfql("MATCH (m {id})-[:T]->(p) RETURN p") lowered to named nodes + label__ filters + a rows/projection pipeline that scanned all edges/nodes and paid ~30ms of rows-pivot machinery even for a single-row seeded result. New _execute_seeded_typed_hop_fast_path (gfql_unified, alongside the existing single-hop-aggregate / two-hop-count fast paths): detects the seeded typed 1-hop with a single whole-row node RETURN, then: - _seeded_typed_return_dst_pandas (chain.py): numpy dst-only extraction — resolves label__X->type, id-first successive subset of the seed (later filters index the tiny survivor rows via .iloc so object columns are never fully materialized), seed-reduces edges before the typed-edge scan, gathers + filters the dst nodes in one node pass; - a lean apply_result_projection on just the p rows (exact column-order/flatten semantics, no rows-pivot). Gated pandas-only, forward seeded shape (RETURN alias == dst node, seed on src); reverse / multi-alias / field / non-scalar returns fall through to the full path. Parity byte-identical fast-vs-full across RETURN p (labelled/unlabelled dst), no-match, and the bail shapes. Measured dgx (26.02-gfql-polars, 50k/200k, warm-median): full 38.2ms -> fast 0.847 ms (45.2x). GATE MET (<=1ms) on the seeded cypher string. 57 chain tests pass, mypy clean. Also lands the native lever-3 helper (_seeded_typed_hop_numpy_pandas + the _try_chain_fast_path seed-first typed-edge path) this builds on: native seeded typed hop 29.1 -> 0.553 ms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
…1755) Re-verify the seeded-Cypher tax fast path against an INDEPENDENT oracle (creators hand-computed from raw edge/node frames), not merely fast==full, since the full path could share a bug. Native returns {seed} u creators (both bound endpoints); cypher RETURN p returns creators only. Decline coverage: two-hop, predicate node filter, predicate edge filter, undirected -> fast path declines and the full path result is byte-identical. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
…1755) The cypher seeded fast path gated on isinstance(e1, ASTEdge) but not on the edge being a genuine single hop. A variable-length edge (-[*1..2]->) is one ASTEdge spanning multiple hops, so the 1-hop seed reduction silently truncated it — MATCH (a {id})-[*1..2]->(b) RETURN b returned only the 1-hop neighbors, breaking pandas-vs-polars parity (TestVarlenAliasHopGate). Gate on e1.is_simple_single_hop() (same canonical check the native fast path uses; also rejects hop labels, output slicing, fixed-point). Adds varlen decline coverage to the fast-path parity suite. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
Annotate _seeded_typed_hop_numpy_pandas / _seeded_typed_return_dst_pandas (Plottable/ASTNode/ASTEdge params, Optional[Plottable] and Optional[Tuple[DataFrameT, DataFrameT]] returns) so their bodies are type-checked, and guard the src/dst/node bindings against None at the _try_chain_fast_path call site. mypy --config-file mypy.ini clean on both changed files (remaining errors are pre-existing polars-lazy, untouched). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
…#1755) Review feedback on the seeded typed-hop fast path: - direction: str -> Direction (the canonical Literal from ast.py) in both helpers. - Make the helpers ENGINE-GENERIC (pandas + cuDF): _seeded_typed_hop_numpy_pandas -> _seeded_typed_hop_pandas_cudf and _seeded_typed_return_dst_pandas -> _seeded_typed_return_dst_pandas_cudf, rewritten to use only the shared pandas/cuDF DataFrame API (no numpy array drops), so the same body runs on both engines. Relax the native (_try_chain_fast_path) and cypher (_execute_seeded_typed_hop_fast_path) gates to Engine.PANDAS|CUDF. - Replace dynamic getattr with static attribute access in gfql_unified (projection.table/.columns, proj_cols[0].kind, n2._name, e1.direction, base_graph._node/_source/_destination) now that types are statically known. - Drop the dead _SEEDED_FASTPATH_GUARD marker (never set; vestigial from the earlier re-run design) — removes the last getattr. Pandas parity + independent-oracle tests still byte-identical (19 pass); mypy clean on both files (only pre-existing polars-lazy errors remain). cuDF parity verified separately on GPU. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
…1755) Answers the coverage question: broaden ENGAGED-shape verification beyond the few hardcoded cases. Adds a 7-shape × {pandas, cuDF} matrix asserting fast-on == fast-off byte-identical (native typed/untyped/reverse/type-src + cypher labelled/unlabelled/no-match), plus an independent-oracle check per engine. cuDF cases importorskip locally, run on GPU/CI. Complements the ~1114 implicit decline-path exercises the cypher conformance suite already drives through the dispatch (real queries that fall through + still return correct values). Also makes _canon_nodes engine-aware (cuDF/polars -> pandas via to_pandas). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
lmeyerov
force-pushed
the
perf/gfql-seeded-typed-hop-fastpath-1755
branch
from
July 21, 2026 20:57
855123a to
dec4e4a
Compare
Contributor
Author
… semantics (#1755) Review-skill wave findings, all empirically confirmed then fixed: - decline under policy (prechain/postchain/postload hooks were silently skipped) - decline on same-path WHERE (cross-alias predicates were silently dropped) - decline on OPTIONAL MATCH empty_result_row (null row became empty frame) - decline on carried reentry seeds (start_nodes ignored -> seed silently widened) - label__X resolution now mirrors resolve_filter_column exactly (list-'labels' column takes precedence; edge frames decline) via shared _seeded_scalar_filters (also dedups the twin _scalars/_sc closures) - membership sets dropna()'d: NaN ids/endpoints never link (pandas .isin matches NaN<->NaN; the full pipeline's joins never join null keys) - 'byte-identical' claims corrected to value-identical (row order/index may differ) - 6 regression tests pinning each gate (TestFastPathSideChannelGates) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Closes the seeded-Cypher abstraction tax (#1755): a seeded typed 1-hop
query —
MATCH (m {id})-[:T]->(p) RETURN p, and the native chain equivalent[n({id}), e(edge_match), n({type})]— spent ~20–38 ms in chain machinery(two-pass combine + full-column
filter_by_dicttype scans + rows-pivotprojection) despite the hop itself being sub-ms. This adds a pandas-only fast
path that numpy-reduces the graph to the seed's 1-hop neighborhood, landing the
same query in ≤1 ms (measured 0.847 ms on a 50k-Person / 200k-Message
graph; native hop 0.553 ms).
Two layers:
_seeded_typed_hop_numpy_pandas, wired into_try_chain_fast_path):scalar-filtered seeded typed 1-hop → a handful of numpy passes.
_execute_seeded_typed_hop_fast_pathin gfql_unified): the loweredMATCH (m {id})-[:T]->(p) RETURN pwhole-row RETURN surface.Both are byte-identical to the full path by construction and fall through
(return
None) for anything outside the covered shape.Correctness / parity
labelled/unlabelled RETURN, no-match, and native forward/reverse/typed shapes.
hand-computed from the raw edge/node frames — not merely fast==full, since
the full path could share a bug. Native returns
{seed} ∪ creators(both boundendpoints); cypher
RETURN preturns creators only.(seed on return node), predicate node/edge filters, undirected, multi-hop, and
variable-length hops all fall through to the full path (asserted, and the
full-path result asserted byte-identical).
compute/gfql+test_chainsuite = 77 failed /5090 passed; clean-master baseline = 77 failed / 5069 passed (identical failure
set — all pre-existing local polars-drift / GPU-lib). +21 net new passing tests.
Notes
change; verified no regression).
mypy --config-file mypy.iniclean on both changed source files (helpers aretyped with
Plottable/ASTNode/ASTEdge+Optionalreturns).accepted variable-length edges (one
ASTEdge, multiple hops) and silentlytruncated them — now gated on
e1.is_simple_single_hop().🤖 Generated with Claude Code